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ABSTRACT 

An approach to a cost analysis of pilot, prototype, model and operational 
instructional management systems (IMS) is described. Five classes of 
systems (conceptual, pilot, prototype, model and operational) are defined 
and a system development process relating the classes of systems is 
presented. IMS functions are discussed in terms of five subsystems: 
data generating, communications, data processing, information storage, 
and report generation. Cost estimates for the conmunications subsystem 
are included* 
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A SUGGESTED APPROACH FOR A COST ANALYSIS 
OF ALTERNATIVE IMS CONFIGURATIONS 



The following system development process is assumed in discussing 
the suggested approach to cost analysis. The system development process 
involves the creation of five distinctive types of systems. Figure 1 
shows the basic sequence in which these systems are designed and evaluated. 



Figure 1 



Basic Development Sequence 
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A conceptual system exists only on paper or in the mind of the 
system designer. A number of alternative conceptual systems may be 
proposed (conceptual system design) for accomplishing the functional 
specifications of the desired operational system. The evaluation of 
conceptual systems may be done by logical or mathematical analysis. 
Proposed conceptual systems may be rejected either on the basis of 
feasibility or because the estimates of operational system cost are 
excessive. Feasibility ground-rules may vary; the designer may be 
limited to **of f-the-shelf " components or restricted to research and 
development that can be completed within a fixed time period. 

Pilot systems are either components of proposed operating system 
or a simplified version of the operating system which simulates some 
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aspects of the operating system's desired performance. Pilot systems 
may be used to test the feasibility of a conceptual system if logical 
analysis could provide no conclusive evidence or to reduce the uncer- 
tainty in predicting operational system costs. Pilot system evaluations 
are typically "quick and dirty." They rarely involve experimental 
designs and may be conducted in ideal or simulated environments rather 
than the environment in which the operational system must function. 

Model systems are designed to test the effectiveness of alternative 
conceptual systems. Rigorous experimental designs are used to compare 
these systems. This may dictate that the system be tested in ideal or 
controlable settings rather than the environment of the operational 
system. 

Prototype systems resemble operational systems to a greater extent 
than do model systems. They are evaluated in the operational or "natural" 
environment; thus, they should provide better information than a model 
system about the costs and "in-practice" deficiencies of operational 
systems but less information about the validity of theory concerning 
system effectiveness. Prototype systems serve two purposes: (1) they 
are used to demonstrate that the proposed operational system will work 
and provide accurate estimates of operational performance and costs, 
and (2) they can be continuously monotored, evaluated rnd modified to 
improve performance or reduce costs. These two purposes may, at times, 
be in conflict if the flexibility required to allow for frequent and 
economical modifications detracts from the system's ability to demon- 
strate the performance and costs of operational systems. 
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Operational systems function in real-world environments; control of 
these environments is difficult which restricts the use of experimental 
designs in evaluations. Evaluative research is also limited since the 
system tends to have less flexibility than a prototype system; it is 
both more difficult to monotor and more costly to modify than the 
prototype. 

The actual process of developing an operational system is generally 
more complex than a sequential evolutionary chain of a conceptual system 
followed by a pilot system followed by model and/or prototype systems 
and culminating in an operational system* Figure 2 includes the planning 
activities, decision points and feedback paths that are needed to describe 
the system development process. The development of each type of system 
is considered to have design, documentation, and evaluation phases. 

At the initiation of the system development process, it may not be 
feasible to specify all of the system performance requirements for the 
desired operational system^ Pilot and model systems may be needed to 
supply needed information such as "what performance is needed or desirable 
and "what is the maximum permissible system cost?" The initial planning 
activity may involve specifying the performance requirements of a pilot 
system(s) that will generate this information. Model systems may be 
developed so that the desirability of proposed performance specifications 
can be evaluated by means of an experimental design. 

The proposed interpretation of the system development process, as 
depicted in Figure 2, is that development of pilot, model, prototype, 
and operational systems can be done during overlapping time periods. 
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Figure 2 
System Development Process 
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Planning Activity: What types of Systems 
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Pilo^ System 

a. Design 

b. Documentation 
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Model System 

a. Design 

b. Documentation 
c* Evaluation 
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Prototype System 
a* Design 
b. Documentation 
c* Evaluation 



Operational System 

a. Design 

b. Documentation 

c. Evaluation 
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One purpose of the planning activity is to decide whether existing 
performance specifications warrant the expending of resources in a 
development process* 

If the development of a prototype system is initiated before the 
performance specifications for the operational system have been completed, 
it should be designed to have a high degree of flexibility so that system 
performance for a variety of performance specifications can be simulated • 
Attempts to design such a flexible system may be futile and time consuming* 
A series of pilot systems may be more effective* 

A cost analysis can be made for any of the five types of systems 
and at various times during the system development process* Frequent 
review of the projected costs and performance of the operational system 
can aid management in deciding at the earliest date to discontinue 
development or to invest resources in additional pilot or prototype 
systems. Cost analyses of pilot and prototype systems are useful in 
setting R&D priorities. The projected cost-benefits of the operational 
system justify the development of prototype systems* The entire 
development process may be viewed, in a cost analysis sense, as a 
systematic procedure for reducing the uncertainty in predicting the 
costs and performance of operational systems* 

Configurations 

Several hardware- software configurations have been proposed as 
possible operational instructional management systems* The potential 
software variations are infinite; only hardware configurations which 
are grossly different will be considered here* Three distinctly different 
configurations are: 



6 



(1) An optical scanner inputs data directly to a time-share, third- 
generation computer via phone lines with reports printed by 
teletypes located in participating schools, 

(2) A PDP-8 computer is used as a data concentrator, receiving 
remote input from a number of optical scanners. Data are then 
transmitted via high-speed phone line to a batch-mode computer. 
Report information is returned to the data concentrator and 
later relayed to schools and printed by a teletype. 

(3) The PDP-8 computer, in addition to operating as a data concen- 
trator, performs some or all of the report generation functions 
of the IMS computer. The batch mode computer, if required at 
all, uses no remote, high-speed input. 

Judging the feasibility and cost-effectiveness of these three 
hardware configurations requires detailed specification of the desired 
performance of the operational system. 

System Characteristics 

There are a number of characteristics of any system which greatly 
effect the costs of the various hardware configurations. A partial list- 
ing which is only suggestive of the total includes: 

(1) The number of instructional systems, classes, and grade levels 
reporting input. 

(2) The frequency, diversity, and quantity of data as input. 

(3) The number and frequency of reports as well as the diversity of 
formats . 

{h) The nature of output other than reports, e.g., generated material. 
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(5) The required turnaround time for a given input /output loop, 

(6) The ability to provide output on demand 

IMS Version 3 

In order to provide a vehicle for the suggested approach to cost 
analysis, the characteristics of IMS Version 3 will be used. IMS Version 
3, although capable of handling other instructional systems and grade 
levels, will be tested initially with a maximum of 6 instructional 
systems at 2 grade levels. Input consists of pupil responses to 
Criterion Exercises and mid-program and end-of-program tests. Output 
consists in formatted reports of pupil scores by outcome, for class or 
group, as well as averages for class or group by outcome. Output on 
demand is not provided. It has the following subsystems: 

(1) Data Generating 

(2) Communications 

(3) Data Processing 

ik) Information Storage 
(5) Report Generation 

IMS Version 3 will be tested in a number of schools during the 
71-72 school year. The data generating subsystem will consist of Unit 
Criterion Elxercises for SWRL instructional programs; the 690-21 DTS as 
described in Reference 1 will serve as the communications subsystem; 
the data will be processed by scoring the test and storing the test 
scores as a longitudinal data base; test scores will be returned to the 
teacher in the form of readable reports. 
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Cost Analysis for Unspecified Performance Requirements 

Cost analysis, particularly the predicting of operational system 
costs, is greatly complicated when in the early developmental stages. 
It may be possible, however, to determine empirically the maximum cost 
that is permissible for a system that performs a certain function. This 
is frequently the approach taken in developing a product which must 
sell for less than some price figure if it is to be marketable. It 
may be possible to estimate how much a school system is willing to pay 
per pupil for various Ilfi functions. Knowledge of permissible costs 
can be of great value in judging whether proposed systems are feasible 
or whether development is warranted. 

It may be desirable to develop a simulation model which will compute 
the costs for alternative hardware- software configurations for specified 
IMS capabilities. Such a model would facilitate projections of opera- 
tional system performance from pilot and prototype test data and make 
it less important that the prototype system closely resemble operational 
systems* The inputs required for such a simulation model would, in 
addition, aid in identifying the requirements for pilot and prototype 
systems and the questions which must be answered in evaluation phases. 
This simulation model should be able to compute the costs of proposed 
pilot, prototype, and model systems in addition to projections of 
operational system costs. 
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Current Cost Esimtates 

Cost estimates for the communications sul)system for IMIS have been 
rade. An additional 4K of core and a larger disc than the current SWRL 
PDP-8 configuration were assumed to give the system the capability for 
testing the third proposed IMS configuration* The data generating, data 
processing and information storage costs are not included since the 
information required to make reasonable estimates of these costs is not 
available at this time. 

The communications subsystem considered consists of the following 
components: 

1. Model 12 Op-Scan input devices 

2. Teletype Corporation (ASR 33) teletypes with send and receive 
capability 

3. PDP-8/E Computer with 10 low-speed I/O ports and one high-speed 
I/O 

4. Telephone line data links with Bell 103A full duplex modems 
Table 1 contains estimates of the number of pupil tests that can 

be processed by the communication system in a working day. Computations 
were made for two standard input forms. The first form is a student 
test sheet which contains six multiple choice questions. The second 
form is a separate answer sheet containing the answers from four, six- 
question test sheets. The following assumptions were made about scanner 
performance: 

1. The document transport time for a single 8^ X 11 sheet is 
5.8 seconds. 
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2« Each seat! cycle takes .2 seconds* 

3« In the Image mode, 8 eleven^blt characters are transmitted for 

each line scanned, 
4« In the darkest mark mode, 1 eleven-bit character Is transmitted 

for each line scanned. 
5. The total time required to process a sheet Is the sum of: 



The total system capacity computations are based on the assumption 
that the "optimal" number of scanners are employed. If fewer than the 
"optimal" number are used for the existing average demand usage, the 
system's capacity is reduced. 

Table 2 gives costs estimates for the equipment required, personnel, 
maintenance, and operating costs. Equipment costs are amortized over 
five years. It Is assumed that all phone line charges for the data 
links between schools and the PDP-8 are for local, unlimited service, 
private lines. The following chart Illustrates the additional costs 
Incurred for schools outside the local calling area of the PDP-8 
Installation. These costs apply to Los Angeles County for .5 average 
demand scanner utilization. 



(a) transport time 



(b) scan time 



(c) transmission time 



Additional Message Units for 
3 minute call 



Annual Cost 



2 



776. 



3 



1470. 



6 



2920. 
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Table 3 gives the per pupil cost (for a single subject area) of 
the communication system for four average demand levels. If fewer than 
the "optimal" number of input units are employed or if inputs are 
outside the local calling area, the per-pupil costs are greater. These 
are lower bounds on per pupil costs. It is assumed that the average 
number of tests given each student is 18 for a single subject area. 
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Table 2 

IMS Communication System Costs 



I, Data Concentrator Costa 






Annual 
Cost 


A,- 


Equipment 

PDP-8/E Unit with 12K core, 










Disc with 831K words, and 2 Magnetic Tape drives* 




56,940, 


11,390, 




High-Speed Port for IMS computer Input, 




1,500. 


300, 




Low- Speed Input ports, 3 units of 4 ports each. 




6,000, 


1,200, 




Data Concentrator modems with rotary. 




5,000. 


1,000, 




Operating Costs 










Coamunlcatlon Costs, PDP-8 to 1108, 


$65 


a month 


6,500, 




Personnel Cost to Operate Scanners, 


$3 


a hour 


20,000, 




Personnel Cost to Operate Data Concentrator 






10,000, 




Operating and Maintenance for Data Concentrator 






5,000, 


Sub-Total 


55,390, 


[!• Cost for Each Scanner-Teletype I/O 








A, 


Equipment 










Model 12 Optical Scanner (Duplex with auto, feed) 




3,700, 


740, 




Teletype, Recelve-Send 




600, 


120, 




Modem 




350. 


70, 




Operating Costs 










Local Telephone line charges 


$5 


a month 


50, 




Teletype maintenance 


$22 


a month 


220, 




Scanner maintenance 


$25 


a month 


250, 


Sub-Total 


1,450. 
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